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Colleges and universities recognize that one of the primary goals of higher education is to promote 
students’ ability to think critically. Using data from the Wabash National Study of Liberal Arts 
Education (WNS), this study examined the relationship between faculty teaching practices and the 
development of students’ critical thinking skills, specifically the differences between students’ self- 
report and the direct assessment (i.e., CAAP) of critical thinking. The results from multinomial 
logistic regression and OLS regression analyses showed that asking challenging questions increased 
both students’ self-reported and the directly measured critical thinking abilities. Interpreting abstract 
concepts as well as giving well-organized presentation increased students’ self-reported gains in 
critical thinking; however, these same practices did not significantly impact their CAAP scores. 
Inconsistent with previous literature, class presentations as well as group discussions decreased 
either students’ self-reported or directly assessed critical thinking abilities. These findings can guide 
faculty teaching practices to foster critical thinking for first-year college students. 


Colleges and universities have long recognized that 
one of the primary goals of higher education is to 
promote students’ ability to think critically (Astin, 
1993; Gellin, 2003; McMillan, 1987). Critical thinking 
is a widely used term that includes skills in identifying, 
analyzing, synthesizing, and evaluating information to 
make informed decisions, and the disposition to apply 
these skills (Ennis, 1993; Giancarlo & Facione, 2001; 
Halpern, 1993; Paul, 1993). Although there is 
considerable debate over the definition and the 
elements of critical thinking, critical thinking has been 
listed as a top priority for undergraduate teaching and 
learning (Astin, 1993; Gellin, 2003; McMillan, 1987). 
Despite the sustained interests in fostering critical 
thinking in higher education, there is evidence that 
college graduates lack critical thinking and problem 
solving skills needed in today’s workplaces (U.S. 
Department of Education, 2006). This discrepancy may 
be addressed, in part, by the fact that those teaching 
critical thinking at the college level do not fully 
understand how to effectively teach these skills and are 
unable to transfer critical thinking knowledge into their 
classrooms (Paul, Elder, & Bartell, 1997). 

Literature Review 

Instructional Practices Facilitating Critical 
Thinking Skills 

Previous research has demonstrated how particular 
formal and informal instructional practices facilitate the 
development of critical thinking skills among 
undergraduates. These practices include such broad 
categories as active learning (Astin, 1993; Kuh, Pace, & 
Vesper, 1997; Pascarella & Terenzini, 1991), teacher 
clarity and feedback (Cabrera, Colbeck, & Terenzini, 
2001; McKeachie, 1990), faculty interactions in and out 


of the classroom (Pascarella & Terenzini, 1991; 
Terenzini, Springer, Pascarella, & Nora, 1995), and 
collaborative learning (Pascarella & Terenzini, 1991; 
Terenzini et ah, 1995). Elowever, research on specific 
instructor-driven instructional practices that affect 
students’ critical thinking is limited. One of these rare 
attempts to study the effect of classroom instruction on 
critical thinking development was conducted by Smith 
(1977). Using direct measures for assessing critical 
thinking, Smith found three kinds of instructor- 
influenced classroom interactions to be consistently and 
positively related to gains in critical thinking: the extent 
to which faculty members encouraged, praised, or used 
student ideas; the amount and cognitive level of student 
participation in class; and the amount of interaction 
among students in a course. Following this attempt, 
Terenzini et al. (1995) also conducted one of the few 
studies on growth in critical thinking ability that 
simultaneously examined the effects of instructional 
methods, particularly student relationships with faculty 
and instructor effectiveness in different types of 
courses. However, after controlling for precollege 
critical thinking ability, none of these variables was 
significant. 

Course assignments and exams are commonly used 
to foster students’ ability to think critically. If 
instructors can engage students in coursework by using 
appropriate instructional methods, students will 
improve their critical thinking skills. In other words, 
what students do for courses matters more than how 
instructors teach courses. In his monumental study 
exploring student involvement in higher education, 
Astin (1993) found assignments such as giving class 
presentations and critiquing papers were positively 
related to students’ self-reported growth in critical 
thinking. Using the data derived from the Cooperative 
Institutional Research Program (CIRP) 1989 follow-up 
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survey, Tsui (1999) investigated the effect of 
instructional techniques on students’ self-reported 
growth in critical thinking. She found that self-assessed 
growth in critical thinking is positively related to such 
instructional factors as having a paper critiqued by an 
instructor, conducting independent research, working 
on a group project, giving a class presentation, and 
taking essay exams; negatively related to this outcome 
was taking multiple-choice exams. Based on these 
findings, her conclusion was that the characteristics of 
the instructional methods that are effective for critical 
thinking development are asking students to construct 
responses or answers to a question, problem, or 
challenge rather than merely to memorize, recognize, 
and select correct answers from among provided 
possible responses. More recently, Tsui (2002) 
conducted a qualitative case study of four higher 
education institutions to explore the contextual factors 
that could affect students’ growth in critical thinking. 
Through interviews and class observations, she found 
that the amount of writing and the nature of the writing 
assignment seemed to matter, such that assignments 
demanding more analysis and less description were 
conducive to promoting critical thinking. Furthermore, 
feedback on one’s writing may further facilitate critical 
thinking, especially if it involves rewriting an 
assignment. According to Tsui (2002), rewriting 
challenged students to utilize others’ feedback to refine 
their own, thus requiring an additional step and 
facilitating critical thinking. 

Coupled with rare efforts that empirically examine 
the effect of particular instructional methods and course 
assignments on critical thinking, some theoretical and 
argumentative writings suggest specific teaching 
methods to foster higher order thinking skills based on 
authors’ teaching experiences. For example, Mills 
(1998) suggested provocation as a method to foster 
higher order thinking in classroom environment. Citing 
his own experience of teaching philosophy using this 
method as an example, he argues the use of challenging 
questions and statements can promote students’ active 
participation and learning, which can result in the 
development of abstract thinking. Owens (2007) argued 
for the importance of students’ self-critique in class for 
promoting higher order thinking abilities such as 
thinking critically, analyzing arguments, and reflecting 
on one’s own assumptions. Although Mills and Owen 
highlight specific teaching strategies to foster critical 
thinking, past research has rarely attempted to test the 
effect of provocation or self-critique on the 
development of critical thinking. 

To summarize, several course-related activities or 
tasks have been identified as significant factors in 
promoting critical thinking: giving a class presentation, 
conducting a group project, writing assignment and 
exams, and conducting independent research, as well as 
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the nature of the task or assignment. In other words, 
assignments or tasks that require more thinking or 
analyzing rather than retrieving or describing (i.e., 
assignments higher on Bloom’s Taxonomy of the 
Cognitive Domain [Anderson, 2001; Bloom, Englehart, 
Furst, Hill, & Krathwohl, 1956]) are effective for 
enhancing critical thinking. Despite these recent 
attempts to unveil the nature of tasks or assignments 
that are specifically helpful for the development of 
critical thinking, the studies that examined the effects of 
instructor-driven teaching practices (e.g., Socratic 
methods, pushing students to apply concepts learned) 
are still rare. 

Defining and Measuring Critical Thinking Skills 

Teaching critical thinking skills to college 
students is complicated partly due to the disagreement 
over the definition and components of critical thinking 
(e.g., see Ennis’ [1962] 12 specific “aspects”; 

Facione’s [1990] 5 “dispositions”; Paul, Binker, 
Jensen, and Kreklau’s [1990] 35 “dimensions”; and 
Clark and Biddle’s [1993] 4 “processes”). Authors 
who have developed the definition and the assessment 
of critical thinking touched on slightly different 
aspects of thinking abilities and highlighted one aspect 
more than the others. For example, Ennis (1962) 
emphasizes the element of making informed decisions 
whereas Facione and Facione (1992) highlight the 
disposition to continuously use critical thinking skills. 
Recently, authors have tried to define the concept of 
critical thinking by discriminating it from other 
similar constructs such as problem solving, reflective 
evaluation and creative thinking. Problem solving is 
most frequently confused with critical thinking 
because both constructs focused on solving the 
problems at hand via analytical thinking and 
reasoning. Bailin, Case, Coombs, and Daniels (1999) 
describe problem solving and decision making as 
arenas in which critical thinking should take place 
rather than as other kinds of thinking to be contrasted 
with critical thinking. In other words, critical thinking 
encompasses the process of solving problem and 
making decisions with specific skills and dispositions. 
Although Marzano et al. (1988) claim that creative 
thinking and critical thinking are similar and 
overlapping constructs, Paul (1993) differentiates 
creative thinking from critical thinking, but highlights 
that both constructs are not mutually exclusive, and 
share some skills and abilities. Ennis (1993), however, 
clearly states that critical thinking does not encompass 
creative thinking. According to Ennis (1993), creative 
thinking is generating new ideas or arguments, 
whereas critical thinking is analyzing or reasoning the 
pre-existing arguments, and forming informed 
decisions or views based on this reasoning. 
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As varied the constructs of critical thinking are, so 
are the measures for critical thinking. Disagreement in 
the construct of critical thinking generates several 
competing measures for critical thinking. To date, the 
most common measures of critical thinking are skills 
tests such as the California Critical Thinking Skills Test 
(Facione, 1990), the Collegiate Assessment of 
Academic Proficienty (CAAP; ACT, 2000), and 
Watson-Glaser Critical Thinking Assessment (Watson 
& Glaser, 1952). Facione and Facione (1992) 
developed an additional measure for measuring critical 
thinking dispositions and attitudes called the California 
Critical Thinking Dispositions Inventory (CCTDI). 
Although these tests share some common features, they 
differ in terms of theoretical bases and may measure 
slightly different constructs. Although some authors 
argue the advantage and accuracy of using direct 
measures of critical thinking, others use self-report 
measures because they are more efficient. Self-report 
has become a widely used method for gathering 
information about college students, particularly their 
attitudes and behaviors. In addition, self-report 
measures generally allow researchers to gather 
information from larger samples (Astin, 1993). 

Researchers generally agree that self-report 
measures are valid within certain limits, but caution 
against using self-report singularly and universally 
(Gonyea, 2005). These cautions are evident for 
measures of academic development (Gonyea, 2005; 
Kuh, 2001; Pike, 1995, 1996). For example, Pike 
(1995) and Astin (1993) found self-report measures to 
be an acceptable proxy for academic development when 
self-report measures and direct assessment measure the 
same constructs, but self-reports cannot replace direct 
assessment measures entirely. Previous literature has 
also reported mixed results of the relationship between 
students’ self-reports and direct measures of critical 
thinking. Some studies reported moderate correlations 
(Pascarella & Terenzini, 1991; Tsui, 1999), while 
Bowman and Seifert (2010) found that small or 
virtually zero correlations between students’ self- 
reports and direct assessments of critical thinking skills. 

Although students’ self-reports can indicate 
cognitive growth, it is possible that students’ self- 
reports and direct assessment measure different aspects 
of critical thinking abilities and skills. The former 
might be related to students’ satisfaction with collegiate 
experiences and students’ own perception of their 
developmental levels, whereas the latter might 
represent an attempted standardized measurement of 
students’ ability levels across disciplines or institutions. 
This possible discrepancy in the measures of critical 
thinking, in turn, highlight the necessity of comparing 
the outcomes derived from different types of measures 
in order to improve the quality of teaching critical 
thinking. 
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The purpose of this study is to investigate the 
effects of various instructor-driven teaching practices 
on the development of students’ critical thinking ability. 
While exploring the effectiveness of specific teaching 
practices, we will also compare students’ self-reported 
measures of critical thinking and direct measures of 
critical thinking in order to generate practical 
implications for teaching and learning. 

The lack of research on classroom instruction as 
well as very vague definitions of instructional practices 
in previous studies brings renewed attention to the 
topic. In other words, past research has not yet fully 
addressed the impact of specific instructional methods 
on students’ critical thinking. Furthermore, what has 
been published cannot answer why particular 
assignments and tasks are more effective than others. 
Therefore, this study will contribute to research on 
critical thinking by specifying various instructor-driven 
teaching practices and course-related task 
characteristics to improve critical thinking, which will 
generate practical implications for professional 
development. Based on the review of the existing 
literature and the unknowns it brought to light, this 
study asks: 

• What instructional practices and assignments 
promoted growth in self-reported critical 
thinking abilities in first-year college students? 

• What instructional practices and assignments 
predict first-year college students’ direct 
measured gains in critical thinking? 

• How do self-report and objective measures of 
gains in critical thinking compare? 

Methods 

Data Source 

The data for this study were derived from the 
Wabash National Study of Liberal Arts Education 
(WNS). King, Kendall Brown, Lindsay, and Van Hecke 
(2007) synthesized the literature on liberal arts 
education and developed a model of liberal arts 
outcomes that embraced seven general dimensions: 
effective reasoning and problem solving, well-being, 
inclination to inquire and lifelong learning, intercultural 
effectiveness, leadership, moral character, and 
integration of learning. For this specific study, we 
focused on the outcome of effective reasoning and 
problem solving which was measured by both students’ 
self-reports of their gains in critical thinking and direct 
assessment of critical thinking. 

The institutions selected for the WNS represent 
differences in college and universities nationwide on a 
variety of characteristics including institutional type 
and control, size, location, and patterns of student 
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residence. As a result, the sample has a total of 19 four- 
year and two-year colleges and universities, from which 
incoming first-year students were selected for 
participation either randomly (for larger institutions) or 
entirely (for smaller institutions). The data were 
collected at the beginning and the end of students’ first 
year, primarily asking their first year experiences at 
college. 

The initial data collection was conducted in the 
early fall of 2006 with 4,501 students. The data 
collection included a WNS precollege survey that 
sought information about student demographic 
characteristics, high school experiences, educational 
degree plans, and the like. Students also completed a 
series of instalments that measured liberal arts 
outcomes. Effective reasoning and problem solving, 
which is conceptualized as "the capacity to make 
reflective judgments; think critically and independently; 
and analyze, synthesize, and evaluate information in 
order to make decisions and solve problems” (King et 
ah, 2007, p. 5), was measured through the critical 
thinking module from the Collegiate Assessment of 
Academic Proficiency (CAAP). 

The Time 2 data collection was conducted in 
spring 2007, resulting in a participation of 3,081 
students with returning rate of 68.5%. For the Time 2 
data collection, two types of data were collected; the 
first was from questionnaire instruments that collected 
extensive information on students’ experience of 
college. Two complementary instruments were used: 
the National Survey of Student Engagement (NSSE; 
Kuh, 2001) and the WNS Student Experiences Survey 
(WSES). These instruments were designed to capture 
student engagement in, or exposure to, empirically 
vetted good practices in undergraduate education. In 
addition, as a part of NSSE, students were asked to 
indicate to what extent they perceived that these 
practices facilitated their learning and development. 
The second type of data collected consisted of posttest 
measures of the instruments measuring dimensions of 
intellectual and personal development using 
instruments noted above. 

For the purpose of present study, we excluded 
students attending two-year institutions (n = 128) 
because of the diversity of educational goals for 
students at two-year institutions. We also selected out 
cases with missing values either on independent or 
dependent variables; this resulted in an analytic sample 
of 1,181 students from 17 institutions. A description of 
our analytical sample is presented in Appendix A. 

Variables 

The dependent variables were two measures that 
assessed students’ development in critical thinking: 
students’ self-reported gains in and direct assessment of 
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students’ critical thinking abilities. Self-reported gains 
were measured by asking students the extent to which 
each teaching practice (e.g., making a class 
presentation, completing a writing assignment) 
contributed to the development of students’ critical 
thinking abilities. The critical thinking module from the 
CAAP was used to assess students’ development in 
critical thinking abilities and skills. The critical thinking 
test is a 40-minute, 32-item instrument designed to 
measured students’ ability to clarify, analyze, evaluate, 
and extend arguments (ACT, 2008). The test consisted 
of four passages in a variety of formats (e.g., case 
studies, debates, dialogues, experimental results, 
statistical arguments, editorials). Each passage 
contained a series of arguments that support a general 
conclusion and a set of multiple-choice test items. 
Scores are calculated from these items and scaled using 
an algorithm devised by ACT to minimize 
measurement errors. The internal consistency 
reliabilities for the CAAP critical thinking test are 
around .85 (ACT, 2008). In addition, Pascarella, Bohr, 
Nora, & Terenzini (1995) reported that in pilot testing 
for the National Study of Student Learning with a 
sample of 30 college students, the critical thinking test 
of CAAP correlates .75 with the total score of Watson- 
Glaser Critical Thinking Appraisal, one of the most 
widely used critical thinking skills assessments. 

Students’ development in critical thinking abilities 
was modeled as a function of individual characteristics, 
institutional characteristics and instructional practices 
(see Figure 1). Because our major focus was on the 
impact of instructional practices on students’ 

development in critical thinking abilities, we controlled 
for individual and institutional variables that potentially 
confound the results. With regard to individual 
characteristic variables, we included gender, 

race/ethnicity, parental educational attainment 

measured with years of education, parental income, 
precollege academic ability, and academic motivation. 
Precollege academic ability was measured with either 
students’ ACT or Time 1 CAAP test scores. These two 
measures are highly correlated with each other (r = .78), 
so we used only one of them for our analyses in order to 
avoid multicollinearity. We used ACT score as a 
control for estimating self-reported gains, and 
precollege CAAP scores as a control for modeling the 
post-test CAAP test scores. 

In order to control for institutional characteristics, 
we included institutional type based on their Carnegie 
classification, and average class size. With regard to 
institutional type, regional or research universities were 
compared to liberal arts colleges. Average class size of 
each institution was a dummy-coded variable that 
compared large classes (average class size > 20) to 
small classes (average class size < 20). The distinction 
between large and small classes was made based on 
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Figure 1 

Conceptual Model Guiding Effects of Instructional Practices on Critical Thinking 



previous literature that defined small classes as 
classes with less than 20 students (Glass & Smith, 
1979). 

Finally, instructional practice variables consisted of 
a set of classroom instruction variables, frequencies of 
different types of class-related tasks, and the 
characteristics of these course-related tasks. All of these 
measures on instructional practices were derived either 
from the WNS student survey or the NSSE survey to 
measure students’ perception of how often these 
instructional practices were implemented during their 
courses. For example, students were asked to respond to 
a question like, “In your experience at your institution 
during the current school year, about how often have 
you received prompt written or oral feedback from 
faculty on your academic performance (Faculty 
feedback)” with 4 or 5 point Likcrt-type scales. A more 
detailed description of each variable is presented in the 
Appendix B. 

Analytical Models 

Students’ self-reported gains in critical thinking 
abilities were analyzed using the multinomial logistic 
regression (MNLR) technique due to the multinomial 
nature of the dependent variable. Students’ self- 
reported gains in the WNS were measured with 4-point 
Likert scale (1: very little to 4: very much). Measures 
with Likert scales are often conceived of as continuous 
outcomes, and analyzed with least squares regression 
technique. Flowever, ordinary least squares (OLS) 
technique could produce biased estimation because 
Likert measures are not actually continuous, but ordinal 
categories. Ordered logistic regression is suggested as a 


technique which produces accurate estimation in case 
of ordinal categorical measures (Peel, Goode, & 
Moutinho, 1998). Flowever, ordered logistic regression 
was not appropriate for the present study due to the 
violation of the parallel regression assumptions (Long, 
1997). Consequently, we adopted multinomial logistic 
regression analysis, and examined differences among 
students who reported different levels of growth in 
critical thinking during their first year at college. For 
the present analysis, two response categories at the 
lower end were combined into “minimum” gains as 
Wald Tests for combining alternatives indicated that 
“very little” and “some” were indistinguishable from 
each other, c2(24, N = 1181) = 23.169 , p = .510. 
Subsequently, “quite a few” and “very much” response 
categories were renamed respectively as “medium” and 
“maximum,” then the utilities of choosing either of 
these categories were compared to the utilities of 
selecting “minimum” response choice. The 
mathematical representation of the MNLR analytic 
model was: 

Uji = XliPij + X 2 ip2j + X 3 iP3j + ejj 

where subscript j denoted the choice (e.g., minimum, 
medium, or maximum) and subscript i denoted the 
individual case. All the Xs on the right-hand side of the 
equation denoted each set of variables. Xj indicated a 
set of individual variables, X 2 denoted a set of 
institutional characteristics, and X 3 represented 
instructional practice variables. In addition, although 
we never observed utility, we inferred from the choices 
people made how they rank some of these alternatives. 
Thus, if an individual reported maximum growth in 
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critical thinking, it must have been the case that the 
utility of selecting that response category exceeded the 
utilities of either choosing “minimum” or “medium” 
response category. In other words, a student chose the 
response that maximized the utility gained from that 
choice. 

Students’ CAAP test scores were analyzed using 
the ordinary least squares (OLS) regression technique 
because the CAAP score is a continuous measure. Our 
OLS model for CAAP scores is mathematically 
represented as follows: 

yCAAP = Xlipi + X 2 ip2 + X 3 ip3 + ej 

Because the study aims to explore the impacts of 
instructional practices on students’ development while 
controlling potential confounding variables, as well as 
to compare different measures on critical thinking, we 
entered all the independent variables into the regression 
at the same time and focused more on comparing OLS 
with MNLR results. 

Results 

Students’ Self-Reported Gains in Critical Thinking 

The results of multinomial logistic model (MNLM) 
on students’ self-reported growth in critical thinking are 
presented in Table 1; this reports the odds ratios that 
compare the probabilities of either “maximum” or 
“medium” response categories to those of the 
“minimum” gains. None of demographic variables (e.g., 
gender, race/ethnicity) were significantly associated 
with students’ self-reported growth of critical thinking 
skills. Among other background and institutional 
variables, students in research universities were 
approximately twice as likely to report maximum 
growth rather than minimum growth, compared to 
students in liberal arts colleges, odds ratio = 2.182. In 
other words, twice as many students in research 
universities thought that they gained a lot of critical 
thinking abilities during their first year than did 
students at liberal arts colleges. 

Among the five faculty-initiated instructional 
practices, the frequency of faculty asking challenging 
questions in class had a substantial influence on 
students’ self-reported growth in critical thinking. To 
illustrate, for a one unit increase in the frequency of 
faculty asking students challenging questions, the odds 
of students’ self-reporting maximum and medium 
growth increased by 98.0% and 44.9% respectively, as 
compared to the odds of students’ reporting minimum 
gains. In other words, the more students were asked 
challenging questions, the more likely they were to 
report medium or maximum gains rather than minimum 
gains. This may imply that students’ level of self-report 
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growth in critical thinking abilities can be increased by 
the frequent use of challenging questions. 

Being in classes with faculty who frequently 
interpreted abstract concepts for students as well as 
giving well-organized presentation distinguished the 
students who reported maximum growth from those 
reporting minimum growth, but not from those 
reporting medium growth. Thus, these practices can be 
said to contribute to only maximizing the students’ self- 
reported growth in critical thinking abilities. To 
illustrate, the frequency of faculty explaining abstract 
concepts in class significantly increased the odds of 
students reporting maximum gains in critical thinking 
as compared to minimum growth response by 60.2%. 
We also found that as faculty more frequently gave 
well-organized presentations of materials in class, the 
odds of students reporting maximum growth in critical 
thinking ability increased by 47.7% in comparison to 
the odds of students choosing the minimum growth 
response category. 

Some variables representing course-related task 
characteristics had significant impacts on students’ 
perceived growth in critical thinking. Frequent use of 
course-related tasks that required students to integrate 
the ideas from multiple sources contributed to an 
increase in odds of students responding either with the 
maximum or medium response category compared to 
the odds of minimum growth. In fact, for a one unit 
increase in the frequency of this type of task, the odds 
of students’ reporting either medium or maximum 
growth in critical thinking ability became larger than 
the odds of minimum growth response by 42.5% and 
129.7% respectively. 

Only two course-related task characteristics 
differentiated students reporting maximum growth from 
those reporting minimum growth. More frequent 
assignments that required students to compare and 
contrast topics or ideas from a course increased the 
odds of students’ reporting maximum growth by 45.4%, 
as compared to the odds of students reporting minimal 
growth in critical thinking. In addition, for a unit 
increase in the frequency of faculty feedback on 
students’ assignments or academic performance, the 
odds of maximum response increased by 71.3% over 
the odds of students’ self-reporting minimum growth in 
critical thinking. These results indicate that instructors 
can maximize students’ perceived gains in critical 
thinking by giving more assignments or exams that 
require students to compare and contrast diverse 
perspectives as well as by providing more frequent 
feedback on these assignments. 

By contrast, frequency of giving presentations was 
significant but negatively associated with students’ self- 
reported gains in critical thinking abilities. As the 
frequency of this assignment increased, the odds of 
students reporting maximum gains decreased by 35.6% 
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Table 1 


MNLM Results of Students ’ Self-Reported Growth in Critical Thinking (odds ratios) 


Outcome: Self-reported growth in critical thinking 

Medium 

Maximum 
vs. Minimum 

Controls: Background Characteristics 



Gender (Female) 

1.250 

1.210 

Race/Ethnicity (Non-white) 

1.493 

1.631 

Mother’s education 

0.993 

1.021 

Father’s education 

1.066 

1.025 

Parental income 

1.013 

1.006 

Academic motivation 

0.967 

1.062 

Precollege academic ability (ACT) 

0.988 

0.995 

Controls: Institutional Variables 



Research university 1 

1.547 

2.182 * 

Regional university 1 

0.746 

0.643 

Large class size (> 15) 

0.850 

0.625 

Variable of Interest: Faculty-Initiated Instructional Practices 



Asking challenging question 

1.449 * 

1.980 *** 

Asking students to apply concepts 

1.089 

1.158 

Asking students to defend point of view 

1.021 

1.115 

Interpreting abstract concepts 

1.158 

1.602 ** 

Well-organized presentation 

1.034 

1.477 * 

Variable of Interest: Course-Related Tasks 



Writing 

1.181 

1.013 

Problem solving 

1.108 

1.162 

Class presentation 2 

0.807 

0.644 * 

Group project 2 

0.982 

0.852 

Variable of Interest: Task Characteristics 



Application of concepts 

0.960 

1.029 

Compare & contrast 

1.275 

1.454 

Defend point of view 

0.962 

1.235 

Integrate ideas 2 

1.425 * 

2.297 *** 

Faculty feedback 2 

1.132 

1.713 “* 

Number of Cases 

1,181.0 


Log likelihood 

-947.37673 


Log likelihood ratio ^2 

374.33 *** 


Pseudo R 2 

0.165 



Note. *** p < .001; ** p < .01; * p < .05 

1 Compared to liberal arts university 

2 Items are derived from NSSE survey; all the others are from WNS student survey 


compared to the odds of reporting minimum gains. In 
other words, students who more frequently gave and 
listened to class presentations tended to perceive less 
growth in critical thinking. This result contradicted 
previous studies which highlight the importance of 
active learning pedagogies, such as giving class 
presentations and engaging in small group discussions 
as promoting students’ critical thinking skills (e.g., 
Pascarella & Terenzini, 1991; Tsui, 1999). 

Students’ Growth in Critical Thinking as Measured 
by CAAP 

Table 2 presents the results of OLS regression on 
the direct assessment of students’ gains in critical 
thinking skills. Unlike the findings of MNLR, a number 


of background and institutional variables significantly 
affected students’ critical thinking ability. Among 
individual characteristics, academic motivation as well 
as pre-college critical thinking ability positively 
affected students’ critical thinking ability. Students’ 
pre-college critical thinking ability was the strongest 
factor predicting their critical thinking ability during 
their first year. Among institutional variables, there was 
a significant difference between students enrolled at 
regional universities and those in liberal arts colleges, 
with liberal arts college students scoring significantly 
higher on CAAP test than those enrolled at regional 
universities. 

Of instructor-initiated teaching practices, asking 
challenging questions in class had a significant and 
positive impact on students’ gains in critical thinking 
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Table 2 


OLS Regression Results of Students ’ Post-Test Critical Thinking (CAAP) 


Outcome: Post-Test Critical Thinking 

B 

OLS Results 

3 

S.E. 

Controls: Background characteristics 




Gender (Female) 

0.079 

0.038 

0.209 

Race/Ethnicity (Non-white) 

-0.440 

-0.188 

0.245 

Mother’s education 

-0.018 

-0.041 

0.053 

Father’s education 

0.016 

0.042 

0.047 

Parental income 

0.016 

0.031 

0.060 

Academic motivation 

0.244 ’ 

0.240 

0.107 

Pre-test critical thinking (pre-CAAP scores) 

0.754 *“ 

0.794 

0.022 

Controls: Institutional variables 




Research university 1 

0.624 

0.286 

0.356 

Regional university 1 

-0.573 * 

-0.227 

0.269 

Large class size ( > 15) 

0.325 

0.161 

0.307 

Variable of Interest: Instructional practices 




Asking challenging questions 

0.547 *** 

0.433 

0.150 

Asking students to apply concepts 

0.012 

0.012 

0.118 

Asking students to defend point of view 

0.120 

0.128 

0.118 

Interpreting abstract concepts 

0.271 

0.217 

0.145 

Well-organized presentation 

0.149 

0.108 

0.154 

Variable of Interest: Course-related tasks 




Writing 

0.220 

0.188 

0.132 

Problem solving 

-0.037 

-0.041 

0.092 

Class presentation 2 

-0.115 

-0.075 

0.165 

Group project 2 

-0.377 ** 

-0.287 

0.134 

Variable of Interest: Task characteristics 




Application of concepts 

-0.148 

-0.156 

0.106 

Compare & contrast 

-0.045 

-0.042 

0.131 

Defend point of view 

-0.131 

-0.147 

0.114 

Integrate ideas 2 

-0.294 

-0.217 

0.150 

Faculty Feedback 2 

-0.011 

-0.009 

0.131 

Number of Cases 

1,181.0 



F (24, 1156) 

0.10 *** 



R 2 

0.6215 



Adjusted R 2 

0.6137 




Note. *** p < .001; ** p < .01; * p < .05 
1 Compared to liberal arts university 

" Items are derived from NSSE survey; all the others are from WNS student survey. 


even after controlling for pre-CAAP scores. In fact, for 
every unit increase in the frequency of instructors 
asking challenging questions in class, students’ scores 
on the CAAP increased by .547 points (b = .547, p < 
.0005). In contrast with the MNLR results, other 
instructional practices were not significantly associated 
with students’ development in critical thinking. 

One type of course-related practices did seem to 
affect students’ gains in critical thinking abilities. The 
frequency of group projects had a significant but 
negative impact on students’ critical thinking skills. 
As the frequency of engaging in group projects 
increased, students’ post-test scores on CAAP 
decreased (b = -.377, p = .005). This result, similar to 
the case of MNLR, contradicted previous literature that 


indicated a positive impact of group work on critical 
thinking abilities (Pascarella et ah, 1995; Tsui, 1999). 
None of the assignment characteristic variables were 
significant for CAAP test scores, which was 
inconsistent with the results from MNLR analysis. 

Discussion 

Our results provided several insights into the 
features of instructional practices that would foster 
first-year college students’ critical thinking abilities. 
First, instructional techniques that not only provoke 
students to think differently (e.g., asking challenging 
questions), but also provide developmental supports 
(e.g., giving well-organized presentations, interpreting 






Shim and Walczak 


abstract concepts) are needed to foster students’ critical 
thinking abilities; this balance is consistent with 
Sanford’s (1966) theory that both support and challenge 
are necessary for growth. Our results indicated that 
posing challenging questions in class improved 
students’ self-reported as well as objectively measured 
critical thinking abilities. Teaching practices such as 
teachers’ frequent explanations of abstract concepts to 
students, as well as well-organized presentations in 
class contributed to increasing self-report growth in 
critical thinking among college freshmen. These two 
teaching practices are slightly different from teachers’ 
asking challenging questions in that the latter stimulates 
students to think creatively or differently, whereas 
explaining abstract concepts as well as giving well- 
organized presentations provides some support. 
Therefore, instructors should use these practices 
complementarily by provoking students through 
questioning that challenges students to view issues from 
different perspectives, and then providing explanations 
to help them understand abstract concepts. In other 
words, the development of critical thinking may require 
instructors to balance cognitive challenge with 
intellectual support, as Sanford (1966) suggested. 

Second, our results indicated that course 
assignments requiring students to compare and contrast, 
and integrate ideas contributed to increasing students’ 
critical thinking abilities. Tasks that require integration 
of ideas, as well as assignments specifying that students 
compare and contrast ideas, require students to gather 
multiple ideas or perspectives, organize them by 
themes, and highlight the differences and 
commonalities among different ideas (Barber, 2008). 
This suggests that the development of critical thinking 
may entail thinking and analyzing multiple ideas 
instead of retrieving and recognizing correct answers, 
which is consistent with what Tsui (2002) found in her 
study. Consequently, instructors need to focus on the 
characteristics of tasks (e.g., whether tasks require 
compare and contrast) rather than types of tasks (e.g., 
writing, class presentation) and to incorporate analytic 
components into each class assignment in order to 
foster students’ critical thinking. 

Third, our findings demonstrated that student- 
implemented course practices such as class presentations 
and group projects did not promote but hindered the 
development of first-year students’ critical thinking 
abilities. These results are inconsistent with previous 
studies that indicated the positive effects of these two 
instructional practices (Astin, 1993; Tsui, 1999). Our 
findings are likely reflective of first-year students’ 
developmental orientation; that is, first-year students are 
often externally-defined and rely on authority figures to 
help them understand the world around them and seek 
approval from others (Baxter Magolda, 2001). For these 
externally-defined students, the act of hearing other 
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students present in class places students in the role of 
expert and therefore contradicts an externally-defined 
epistemological belief. They would prefer to learn 
directly from an instructor rather than from a peer. In a 
similar vein, externally-defined students in the group 
situations may be reluctant to challenge each other and 
share ideas with peers for fear of disrupting the group 
and potentially receiving negative feedback from others. 
The positive effect of faculty giving feedback on 
students’ critical thinking abilities may also reflect first- 
year students’ reliance on authority figures for learning 
critical thinking abilities. In sum, instructors who teach 
first-year students may benefit from attending to 
students’ developmental orientations and restructuring 
student-implemented course practices (e.g., class 
presentation) to meet their developmental capacity. 

Finally, we found some discrepancies between 
students’ self-report and direct assessment of critical 
thinking. For example, frequent use of interpreting 
abstract concepts had a significant, positive effect on 
students’ self-reported gain; however, this was not a 
significant predictor for the increase of CAAP scores. 
One possible explanation for these discrepancies is that 
the direct measure of gains in critical thinking and 
student self-reports of gains in critical thinking measure 
different constructs. For the direct measure, the 
developers of CAAP defined critical thinking, whereas 
each student defined critical thinking for self-report 
measures. Given the breadth of critical thinking 
definitions within the literature, we expect student 
interpretations of critical thinking to be just as varied. 
Although this variation is problematic, the different 
definitions may also encompass a broader critical 
thinking skill set than the CAAP. Therefore, the student- 
identified teaching practices may promote critical 
thinking skills beyond what the CAAP measures, such as 
creativity (Marzano et ah, 1988) or reflective thinking 
(Aretz, Bolen, & Devereux, 1997). 

In addition to variation in the critical thinking 
construct, student self-reports might also reflect student 
satisfaction and student engagement (Bowman & Seifert, 
2010). As Astin (1993) reported, when students believed 
they were learning, they were more satisfied and more 
engaged with their educational experiences. Increased 
satisfaction and particularly engagement in learning 
complemented the pedagogical methods employed by 
faculty to improve critical thinking skills because 
students already held faculty and faculty teaching 
practices in high regard. On the other hand, students who 
were dissatisfied with their college experience were less 
likely to report experiences as having a positive impact 
on their development (Bowman & Seifert,2010). Future 
research may include measures of student satisfaction in 
order to better understand the nature of self-report and 
direct assessments on the development of critical 
thinking. 
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Limitations 

The WNS dataset has several limitations that 
should be kept in mind when interpreting the findings. 
First, although the overall sample includes a broad 
range of different kinds of postsecondary institutions 
from 11 different states, the fact that institutions were 
not selected randomly and that the liberal arts colleges 
were purposely over-sampled means that one cannot 
necessarily generalize the results to the population of all 
four-year institutions in the United States. 

A second limitation is the fact that not all students 
who participated in the first (precollege) data collection 
participated in the second (follow-up) data collection. 
The 68.5% persistence rate in the WNS from the first to 
second data collections is consistent with other large 
longitudinal studies requiring a substantial amount of 
participation in terms of time and intellectual effort 
(e.g., see the National Study of Student Learning, 
Pascarella, Edison, Nora, Hagedorn, & Terenzini, 
1998). However, attrition from the first to second data 
collections is a major, if perhaps unavoidable, 
limitation of the study. In other words, we cannot 
guarantee that those students who dropped out of study 
after the first data collection would have responded in 
the same way as their counterparts who persisted in the 
study from the first to second waves. 

Finally, although we cautiously selected 
instructional methods variables that best captured our 
conceptual model describing the effects of course 
practices on students’ critical thinking abilities, the use 
of secondary data sources can limit the investigation of 
our intended conceptual model. There could be other 
instructional methods commonly practiced in colleges 
to promote the ability of critical thinking other than 
those measured in the first-year of the WNS. 
Consequently, findings should be regarded as a partial 
explanation that addresses the effect of instructional 
practices on students’ growth in critical thinking during 
only their first year at college. 

Implications and Conclusion 

A number of implications for educational research 
and practice can be drawn from this study. First, the 
findings provide practical implications for professional 
training of future faculty members and graduate 
teaching assistants. Instructors and teaching assistants 
need to learn how to organize the class presentations, 
formulate and ask challenging questions in class, give 
clear explanations about abstract concepts, and 
encourage students to apply course concepts for 
effectively teaching critical thinking abilities. Although 
these teaching practices could be effective across fields 
of study, the actual implementation should tailor to the 
characteristics of each field. For example, probing on 


Development of Students’ Critical Thinking Skills 25 


underlying assumptions of an argument would be a 
challenging question in humanities while encouraging 
students to integrate multiple theorems for a problem at 
hand would be challenging in engineering. For this 
reason, professional training would reflect the context 
and the characteristics of each disciplinary field. 

Second, with regard to course design, instructors 
should pay more attention to determining why certain 
assignments or exams are important. This study 
demonstrates that assignments or exams are only 
effective in promoting critical thinking because of 
specific characteristics of those assignments or exams. 
Thus, when instructors design a course, they need to 
determine the specifics and the core elements of each 
assignment. Furthermore, these assignments should 
incorporate the element of extracting diverse ideas from 
multiple sources to foster critical thinking. 

Lastly, if institutions are truly committed to the 
development of students’ critical thinking abilities, 
they need to research environmental factors that either 
facilitate or hinder the development of students’ 
critical thinking skills. From the findings of this study, 
students’ self-reports as well as direct assessments of 
critical thinking differed across different types of 
institutions. Students in research universities tended to 
report maximum growth in critical thinking rather 
than minimum compared to students in liberal arts 
colleges. In a similar vein, students in regional 
universities scored significantly lower on CAAP 
compared to those in liberal arts colleges. Combined 
together, these findings suggest that educational 
practices in research universities or liberal arts 
colleges work more effectively than in regional 
universities. These institutional differences may result 
from different practices implemented in each 
institution or from differential effects of similar 
practices on students attending each institution. 
Whatever the case would be, institutions need to 
research best practices on campus and examine why 
these practices work in their context. These best 
practices, once compiled, could be used for 
professional development purposes. 

This research provides insight into future areas of 
research. As noted above, the relationships between 
self-report measures and direct measures may change 
over time. Future research should include comparisons 
between self-report and direct measures of gains in 
critical thinking at the end of college. In addition, 
comparisons between self-report measures in the first 
year and direct measures at the end of college might 
provide insight into whether there is a time lapse 
between what practices students experienced as 
enhancing critical thinking in the first year and what 
practices actually improve CAAP scores at the end of 
college. In addition, future studies could use multiple 
items to measure students’ self-reported gains in 
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critical thinking ability in order to improve 
measurement validity and reliability. 

Since self-report results largely depend on student 
interpretations, qualitative and mixed methods research 
would provide deeper understanding of the topic. For 
example, student interviews could explore the basis of 
student interpretations of the critical thinking construct, 
as well as gain deeper understanding of the exact faculty 
teaching practices that students identify as important. 
Interviews could also assess the types of challenging 
questions faculty ask, as well why the student believed 
these questions developed critical thinking skills. In 
addition, mixed methods research may help explain the 
contradictory findings about types of assignments by 
providing insight into how students perceive group work 
and class presentations. 

Overall, these findings highlight the importance of 
faculty-driven teaching practices in class, such as asking 
challenging questions or encouraging students to apply 
course concepts to real-world situations. In addition, by 
using specified measures that better capture the actual 
instructional methods used in college classrooms, these 
findings give a more clear and detailed explanation of the 
kinds of teaching practices that make a difference in 
promoting critical thinking. The identification of 
characteristics of course-related tasks that can increase 
students’ perceived gains in critical thinking is another 
important finding of the present study. The study found 
that it is not the type of tasks (e.g., writing reports, class 
presentation), but rather the task demands (e.g., 
assignments asking compare and contrast, assignment 
asking application of concepts) that help students 
improve their critical thinking. Although instructional 
practices and certain assignments can promote students’ 
critical thinking, contextual factors such as institutional 
type may differently shape the effectiveness of these 
instructional methods. Thus, colleges and universities 
need to pay more attention to environmental factors that 
can either facilitate or hinder students’ development of 
critical thinking. 
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Appendix A 
Sample Characteristics 


Variables 


Percent 


Gender 

Female 

63 % 



Male 

37% 


Race 

Non-white 

24% 



White 

76% 


Class size 

Large class (> 20) 

43 % 



Small class 

57% 


Institutional type 

Research 

29% 



Regional 

19% 



Liberal arts 

52% 


Variables 


Mean 

Standard Deviation 

Mother’s Education (in years) 

15.32 

2.25 

Father’s Education (in years) 


15.69 

2.64 

Family income 1 


5.66 

1.90 

Precollege academic ability 2 


27.16 

4.20 

Academic motivation 


.00 

.98 

Pre-test CAAP scores 


64.13 

5.02 

Post-test CAAP scores 


64.68 *** 4 

5.37 

Self-report gains in critical thinking 3 

2.37 

.67 


1 1 if Less than $14,999; 2 if- $24,999; 3 if- $34,999; 4 if- $49,999; 5 if- $74,999; 6 if- $99,999; 7- $199,999; 
8 if - $299,999; 9 if $300,000 or more 

2 ACT score 

3 1 = minimal growth, 2 = medium growth, 3 = maximal growth 

4 The difference between pre-test and post-test CAAP scores is statistically significant, t(1180)=5.2762,/> <.0005 
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Appendix B 

Variable Definitions and Coding Schemes 


Variable 

Description 

Outcome 

Self-reported growth in the ability of 

“Extent to which the institution contributes to the development of 

critical thinking 

students’ critical and analytical thinking; ”1 minimum, 2 medium, 3 
maximum 

CAAP 

Scores computed and scaled using algorithm devised by ACT 

Demographics 

Gender 

1 if female, 0 otherwise 

Race/ethnicity 

1 if white, 0 otherwise 

Mother’s education / 

11 if did not finish high school; 12 if High school graduate; 13 if 

Father’s education 

Attended college but no degree; 

14 if Vocational certificate or 2-year degrees; 

16 if Bachelors or other 4-year degree; 18 if Masters; 

19 if Law school; 20 if Doctorate 

Family income 

1 if Less than $14,999; 2 if- $24,999; 3 if- $34,999; 

4 if- $49,999; 5 if- $74,999; 6 if- $99,999; 7- $199,999; 

8 if - $299,999; 9 if $300,000 or more 

Precollege academic ability 

Common high school achievement metric calculated either from SAT or 
ACT scores 

Academic motivation 

composite mean of four items measuring academic motivation (e.g., 
willingness to work hard to learn even if it won’t lead to a higher grad) 

Institutional Characteristics 

Research university 

1 if research university, 0 otherwise 

Regional university 

1 if regional university, 0 otherwise 

Liberal Arts university 

reference category 

Average class size 

1 if larger than 15, 0 otherwise 

Instructional practices in class 

Asking challenging questions 

5 point scale: 1 = never, to 5 = very often 

Asking students to apply 

5 point scale: 1 = never, to 5 = very often 

Asking students to defend point of view 

5 point scale: 1 = never, to 5 = very often 

Interpreting abstract concepts 

5 point scale: 1 = never, to 5 = very often 

Well-organized presentation 

5 point scale: 1 = never, to 5 = very often 

Assignment Types 

Writing 

5 point scale: 1 = never, to 5 = very often 

Problem solving 

5 point scale: 1 = never, to 5 = very often 

Class presentation 

4 point scale: 1 = never, to 4 = very often 

Group project 

4 point scale: 1 = never, to 4 = very often 

Assignment Characteristics 

Application 

5 point scale: 1 = never, to 5 = very often 

Compare & contrast 

5 point scale: 1 = never, to 5 = very often 

Defend point of view 

5 point scale: 1 = never, to 5 = very often 

Integration of ideas 

4 point scale: 1 = never, to 4 = very often 

Faculty Feedback 

4 point scale: 1 = never, to 4 = very often 


1 Percent of students who respond with each response category 

2 Number of students who respond with each response category 





